Bulletin of Electrical Engineering and Informatics 
Vol. 10, No. 3, June 2021, pp. 1345~1355 
ISSN: 2302-9285, DOI: 10.1159 1/eei.v1013.2955 O 1345 


Smart and real-time door lock system for an elderly user based 


on face recognition 


Rezki Saputra, Nico Surantha 


Department of Computer Science, Binus Graduate Program-Master of Computer Science, Bina Nusantara University, 


Article Info 
Article history: 


Received Nov 4, 2020 
Revised Jan 19, 2021 
Accepted Apr 6, 2021 


Keywords: 


Automation 

Door lock system 

Face recognition 

Local binary pattern histogram 
Real-time 

Smart system 


Jakarta, Indonesia 11480 


ABSTRACT 


Many sophisticated smart door lock systems have been made. Still, most of 
them required the user to use an additional device such as a smartphone, tag, 
smartcard, or accessing some user interfaces, which is complicated to use for 
inexperienced elderly. This condition creates a gap between the elderly and 
technology which makes it difficult for the elderly to accept and use the 
technology. In this paper, we proposed a smart and real-time door lock 
system for an elderly user based on local binary pattern histogram as a face 
recognition algorithm with modular system architecture design. The novelty 
in our proposed system design, it does not require any additional device, it 
does not use any user interface, and the least user participation by automating 
the processes. All the user needs to do just walk toward the door and stand in 
front of it and the door will automatically unlock and locked back after the 
user enters the house and close the door. The system resulted in an accuracy 
of 98%, with an average processing time is 1.449 seconds for the entire 
process. Additional advantages, the system is designed with a modular 


approach that makes it flexible and scalable for further development. 


This is an open access article under the CC BY-SA license. 


Corresponding Author: 


Nico Surantha 

Department of Computer Science 
Bina Nusantara University 
Jakarta, Indonesia 

Email: nico.surantha @ binus.ac.id 


1. INTRODUCTION 

The number of elderly is growing globally. Solutions are needed to cope with the complex care 
demands of aging and to satisfy the needs of elderly persons to live longer in their own homes [1]. 
Technology has been an answer for assisting their life, especially for the ones who have been living alone 
without family to help and take care of them. However, it is essential for the elderly could accept and use the 
given supporting technology. The acceptance model of technology for the elderly are required two necessary 
conditions perceived ease of use and perceived usefulness [2]. 

The door is one of the most critical elements of a house because it is used as an entrance for the 
owner. According to the work from [3], Burglary cases involving the elderly and disabled are increasing 
because there is no proper security system for them in the house. The simplicity of the door lock system 
based on technology can be useful for the elderly and help them by protecting their home from thieves. 
Aging leads to an increased risk of developing chronic diseases, such as Alzheimer's disease [1]. The process 
of unlocking and locking the door of the house could be a problem if they lost the key by accidentally 
dropping or forgot the position where they left the key. The development and the innovations on the 
conventional lock are aimed at decreasing the weaknesses [4]. Digital door lock system equipment that uses 
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digital information such as secret code, semi-conductors, smart cards, and fingerprints as the method for 
authentication instead of the legacy system [5]. 

Many digital door lock systems are made use of a password to access the lock [6], [7], and a secret 
code that is saved in some device such as a smartcard, tag, and smartphone [8], [9]. The lock that can be 
controlled through mobile application [10]-[13]. Those are sophisticated technology design which would not 
be a problem when used by tech-savvy users. But in the hands of elderlies, sophisticated technology will be a 
problem because of the incompatibility [14]-[16], and the adoption will become low, which is caused by 
complicated systems and user interfaces [17]. 

The solution to this problem is to design a sophisticated technology with an automation approach 
that will help the elderly to do their routines [18], [19]. The automation approach will make the elderly used 
sophisticated technology without bothering them to use it. The authentication method that will be used in this 
paper is face recognition. The reason is face recognition system needs the least human cooperation and is 
viable too [20]. Face recognition also appropriate to build an application that will automate entry by 
recognizing the face of the person entering the house [21]. 


2. RELATED WORKS 

Many sophisticated smart door lock systems have been made. Table 1 shows the comparison of the 
previous smart door lock research based on the authentication method, additional device requirement, and 
manual input requirement. The Table contains several types of smart door locks that have been made by 
previous researchers. Most of those systems require a smartphone as an additional device to operate the 
system and to achieve an authenticated condition by the system, the user still has to manually input 
commands to the systems via the interface provided by the smartphone. The problem could arise if a user is 
an elderly person who is not familiar with smartphone technology and its user interface, of course, this will 
make it difficult for them to use. 

Therefore, this research will produce a system that is not only safe with its real-time face recognition 
authentication methods but also easy and convenient to use because it does not require additional devices, 
does not require manual input commands from users, and does not require any user interface skills. The 
entire system of the smart door locks is embedded in the door and operated fully automatically. This is a 
novelty design of a smart door lock system which will be discussed further in this study. 


Table 1. Related works 


Ref No Authentication method Additional device Manual input 

[6] Password matching Keypad Required 

[7] NFC authentication and password matching Smartphone Required 

[8] NFC authentication and face recognition Smartphone Required 

[9] NFC authentication and password matching Smartphone Required 

[10] Mobile phone network authentication Smartphone Required 

[11] Mobile phone Bluetooth authentication Smartphone Required 

[12] Mobile phone Bluetooth authentication Smartphone Required 

[13] Captured image face recognition authentication Smartphone Required 
This research Real-time face recognition Not required Not Required 


3. PROPOSED SYSTEM 

This paper aims to propose easy to use and useful face recognition door lock with an automation 
approach to help the elderly securing their door and without the need for various additional devices to operate 
it. This paper describes the design of a system for supporting the elderly, which may have limited physical or 
cognitive capabilities. The prototype should be working automatically with the least participation from users 
and the user does not need to access any user interface. The prototype will be made of the low-cost hardware 
and modularity approach to make the system scalable with additional functionalities. This paper also 
compares two face recognition algorithm that most suitable for use on the prototype by evaluating the 
performances. 


3.1. Proposed system architecture 

The automatic face recognition door lock will be divided into three modules, movement detection 
module, processor module, and lock control module. This system uses a modular approach, so there will be 
three modules that are mutually integrated and form one system. Figure 1 is the picture of the proposed 
diagram for automatic face recognition door lock. The prototype will be made from low-cost hardware and a 
modularity approach. It consists of three modules that are movement detection, processor, and lock control 
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module. The movement detection module is built from Arduino, passive infrared sensor (PIR sensor), 5 V 
relay, and alternating current (AC) lamp. The processor module is built from Raspberry Pi, Pi camera, and 
magnetic switch. The lock control module is built from metal oxide semiconductor field effect transistor 
(MOSFET) IRF520 and a solenoid lock. 


Processor Module Lock Control Module 


Raspberry Pi 
3 


MOSFET IRF520 


GPIO 
DigstalPin Interface 


R 
Relay 5y Lamp 


, © A 
Ç Pi Camera 
> sem 


) PIR Sensor ss. Solenoid 
Owner Arrival Magnetic Switch 





Figure 1. Block diagram system 


3.2. Proposed system flowchart 

The automatic face recognition door lock starts to work by detecting movement using PIR Sensor. 
After getting input from the PIR sensor, Arduino will turn on the lamp and simultaneously send a message to 
Raspberry Pi to execute the face recognition program. If the detected person is recognized as the owner, 
Raspberry Pi will trigger the solenoid to unlock the door, and then the owner can open the door and enter the 
house. As long as the door is open, the magnetic switch state will become low. Raspberry Pi will monitor this 
state. If the door closed, the magnetic switch state would become high, and Raspberry Pi will stop the trigger 
to the solenoid to lock the door. This sequence will run automatically without the intervention from the 
owner as of the user. Figure 2 1s a complete flowchart of the system. 


3.3. Proposed system use case diagram 

The use case diagram will be explaining how the relationship between actor and system. It will also 
give detailed information about what functions the system has. Figure 3 is the use case diagram of the 
automatic face recognition door lock. 


3.4. Face recognition algorithm-local binary pattern histogram 

The local binary pattern histogram (LBPH) is not only simple and low computation complexity but 
also has a rotational invariance, grayscale invariance, and other significant advantages for face analysis and 
recognition [22]. This makes LBPH can be used on low computational platforms like the Raspberry Pi. The 
first computational process of the LBPH algorithm is to convert the grayscale facial image (dataset) to a 
decimal value. Figure 4 shows the conversion process. The facial image will be divided into nine grid (3x3) 
pixels. Each pixel contains a pixel intensity value with a range of values 0-255. Before converting to a 
decimal value, all grid would be converted to a binary value. Centre pixel value would become the threshold. 
If the value is equal or greater than the threshold, the binary value is 1. If the value is smaller than the 
threshold, the binary value is 0. After this process, the system has a threshold result of LBP. The threshold 
result would be converted into pattern result to a form histogram by multiplying the threshold result to binary 
to decimal conversion mask. Figure 5 shows the process of generating the final histogram by dividing the 
image into an 8x8 grid. Each histogram from each grid contains a decimal value from O to 255. The final 
histogram is the concatenated result from each generated histogram. 
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Figure 3. Use case diagram 
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Figure 5. Local binary pattern histogram 


After getting the histogram of each input image, the algorithm is already trained. Each histogram 
created will be used to represent each image from the training dataset. The face recognition process 
performing by input a new image by generating it using a face detection algorithm from a video in real-time 
and create a histogram that represents the image and calculates the distance between two histograms with a 
trained dataset histogram. The LBPH algorithm used Euclidean distance based on the formula: 





pā X Qhistti — hist2i)2 


i=1 


The formula output is the ID from the image with the closest histogram. The algorithm also returns 
the calculated distance, which can be used as a confidence measurement. As Euclidean measures the gap 
between the images, the smaller the distance, the higher the similarity [23]. 


3.5. Face recognition algorithm-dlib face_recognition 

Dlib library face_recognition consists of some kind of recognition modes such as histogram of 
oriented gradient (HOG), support vector machine (SVM), and convolutional neural network (CNN). Even 
though dlib with HOG only detects a frontal image, and CNN can detect multiple angles, HOG used less 
resource or computational cost than CNN [24]. So, this paper used dlib face_recognition based on HOG 
because of its low computational cost and fit the use case of this research. 

The first computational process of dlib face_recognition is creating face encoding from the facial 
image dataset. Histogram of oriented gradient (HOG) and used for finding a face in dataset image. This 
process is done by calculating the gradient value by using a one-dimensional centered mask with subsequent 
filtering of the color or saturation of an image [25]. After finding the face, the algorithm will use a facial 
landmark estimation algorithm to detect 68 landmarks that are present in the images, and the output is a 
centered facial image, as shown in Figure 6. 





Figure 6. Estimation of a centered facial process using facial landmark 
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After the finished facial landmark, the face will be trained to become a 128-dimensional vector 
space where images of the same person are near to each other, and pictures from different people are far 
apart. Face recognition is performed by mapping the input face to the 128-dimensional vector and checking 
the Euclidean distance. If the Euclidean distance is very small or below the threshold value, the system will 
authenticate the person. Still, if the euclidean distance is significant, the system would not authenticate the 
person. This entire process is shown in Figure 7. 
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Figure 7. Face recognition process using dlib face_recognition 


3.6. Proposed system software architecture 

This paper will be tested two face recognition algorithms on the same hardware architecture in real- 
time conditions. The first algorithm to be tested is the LBPH from the open curriculum vitae (CV) library, 
and the second algorithm to be tested is dlib face_recognition based on the histogram of oriented gradient 
(HOG) from dlib library. Figure 8 is software architecture to use LBPH, and Figure 9 is software architecture 
to use dlib face_recognition. A real-time face detection algorithm to be used for each architecture is the Haar 
cascade classifier, which is based on a viola-jones robust real-time face detection algorithm [26]. This reason 
is that viola-jones face detection is more suitable for real-time face detection, requires less CPU resource, and 
fast to be processed by Raspberry Pi [27], [28]. 
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Figure 8. The software architecture of the LBPH based system 
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Figure 9. The software architecture of the dlib face_recognition-based system 
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4. RESEARCH METHOD 
This section is explained the systematic experimental processes utilized to uncover new information 
for analysis purposes and create a better understanding of the research. 


4.1. Prototype design 

The schematic diagram in Figure 10 shows the overall system design while each module is 
integrated. This prototype design will be tested with two face recognition algorithms. This aims to see 
whether the system can work with several types of algorithms and which face recognition algorithms are 
more suitable for use in this prototype design. 
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Figure 10. Schematic diagram of the prototype design 


4.2. Test scenario 

The testing process will involve four actors. Two actors act as true eve whose face image has been 
used as the dataset and will be authenticated by the system. The other two actors act as a non-owner (false- 
event) whose face image has not been used as the dataset and will not be verified by the system. Dataset 
images are taken separately using the haar cascade algorithm program and converted to grayscale as many as 
30 images. In this testing, there are two parameters to be tested that are accuracy and processing time. 
Accuracy is obtained from the evaluation of the system performance using a confusion matrix, and the 
processing time is the time that takes for the system to process incoming (human) input until the system 
unlocks the door. The calculated processing time interval can be seen in Figure 11. 
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Figure 11. Processing time 


Smart and real-time door lock system for an elderly user based on face recognition (Rezki Saputra) 


1352 O ISSN: 2302-9285 


4.3. Evaluation method 

Evaluation is the process of determining the performance of the designed system. The evaluation 
will be done using the confusion matrix. Figure 12 is the confusion matrix. There are four terms as a 
representative result of the classification process. True positive (TP), while the input is the owner, and the 
system recognizes the input as owner. True negative (TN), while the input is not the owner and the system, 
does not recognize the input as owner. False positive (FP), while the input is not the owner, but the system 
recognizes the input as the owner. False negative (FN) while the input is the owner, but the system does not 
recognize the input as the owner. 


5. RESULT AND DISCUSSION 
This section has explained the implementation, test, and evaluation results of the automatic face 
recognition door lock. 


5.1. System prototype implementation 


Figure 12 shows the prototype of the system hardware its implementation in the testing 
environment. The testing environment is the real place, which is the entrance door of a real house. 
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Figure 12. System prototype and its implementation in the testing environment 


5.2. Functional test 

After installing the system in the testing environment, the system should be passed the functional 
test to determine whether the system is working as planned or not. This test is crucial to do before testing the 
system performance. This test will be done on two face recognition algorithms LBPH and dlib 
face_recognition. The system will be activated and entering idle status, waiting for the owner to walk towards 
the door, as shown in Figure 13. After the owner arrives at the door, the PIR sensor will detect movement, 
and the lights automatically turn on then simultaneously run the face recognition will be carried out. After the 
owner is authenticated, as shown in Figure 14. The owner just needs to turn the door handle to open the door, 
then enter the house. After the owner enters the house and the door is closed, the system will automatically 
lock the door. This process was considered as a true-positive condition and can be concluded the system is 
working as planned. 





(a) (b) (a) (b) 


Figure 13. System functional test, (a) The system is Figure 14. Face recognition output, (a) LBPH face 
idle, (b) The system is activated after detecting recognition, (b) Dlib face recognition 
someone 
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5.3. Performance evaluation 

This section will be discussing how the results of testing accuracy and processing time. Table 2 is 
the confusion matrix for LBPH based system and Table 3 is dlib face_recognition based system. Figure 15 is 
the performance comparison of LBPH and dlib face_recognition. Figure 16 shows the average processing 
time comparison of LBPH and dlib face_recognition. 
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Figure 15. Performance comparison 


The results of the implementation and experimental tests on the prototype, it was found in Figure 16 
that the LBPH has better accuracy than the dlib face_recognition. The face recognition process in testing is 
carried out in real-time, where the input from the system is a real-time video captured by the raspberry pi 
camera. It is converted into a frame, and from that frame, the face detection process is carried out. The face 
detection method is carried out using the haar cascade classifier. Haar cascade classifier results are the face 
that becomes the input system to continue the recognition process using the local binary pattern histogram or 
dlib face_recognition method. 

LBPH algorithm is resistant to differences in expression, lighting, and poses. This makes the LBPH 
is very suitable for use in real-time applications where image input is dynamic [29]. The dlib 
face_recognition algorithm used is based on the histogram of oriented gradient (HOG). HOG is known as an 
algorithm that has resistance to the 2D rotation but has no resistance to lighting. [30] therefore this is a 
drawback of the HOG-based method in this system architecture where the lighting is given at a certain angle, 
and the user's position as the face input does not stand in the same position all the time. So, of course, there 
will be differences in the input frame captured by the system. This is what makes the accuracy of LBPH 
slightly better than dlib face_recognition. In Table 1, dlib face_recognition is superior in detecting positive 
input and but not superior in detecting negative input. Meanwhile, LBPH is not superior in detecting positive 
input, but it is very good at detecting negative input. In the context of the security system, a false positive is 
an error that should not exist on the system because it will allow non-authorized persons to have access due 
to system errors. 

The next thing that was tested in the experiment was processing time. LBPH has a better processing 
time than dlib face_recognition. The last process in face recognition used by both methods is Euclidean 
distance. This process compares the trained features from the image dataset and features from real-time input. 
Dlib face_recognition stores the trained feature with the pickle data type so that the data is encoded, and 
during the face recognition process, the system must decode the pickle file before the distance between the 
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trained feature and the input feature can be calculated. In contrast to LBPH, trained data is stored in the 
YAML format so that no encoding and decoding process is occurred and makes the recognition process faster 
and low computational cost. In this case, there is a trade-off in terms of security trained data and processing 
speed. Based on the use case in this study, the Raspberry Pi, which stores trained data, is designed to work 
standalone and headless so that trained data in formal YAML is still categorized as safe from theft and 
sabotage. 
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Figure 16. Average processing time comparison 


6. CONCLUSION 

The face recognition algorithm tested has good accuracy of 98% for local binary pattern histogram 
and 96% for dlib face_recognition. The average processing time for the algorithms is 1.472 seconds for local 
binary pattern histogram and 2.221 seconds for dlib face_recognition. Based on these data, the local binary 
pattern histogram algorithm has better performance for the novel system architecture that has been designed. 
The research in this paper has succeeded in eliminating locking and unlocking the door activities and remove 
any additional object called the key. The process of locking and unlocking the door is done automatically, 
and the key to the door is the owner's face. This makes these activities require the least user participation. 
The prototype is designed in a modular system that would become flexible and scalable for further 
development. The result of this study can be used to provide technical support to the elderly who are not 
experienced with technology. Future work for this system is by adding automatic door opening and closing 
devices such as electrically controllable motors and hydraulics that makes the system can be used as a 
solution to prevent virus transmission during the COVID-19 Pandemic because users would not need to 
touch the door before entering the house. 
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